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Executive summary 

An equating study was carried out in autumn 2006 by the National 
Foundation for Educational Research (NFER) on behalf of Renaissance 
Learning, to provide validation evidence for the use of the Renaissance 
Star Reading and Star Mathematics tests in English schools. The study 
aimed to investigate the correlation between the Star tests and established 
tests. 

Between 11 and 16 schools were involved in the study, with completed 
Progress in Mathematics paper tests received from 2006 primary pupils 
and 883 secondary pupils. Suffolk Reading Scale 2 tests were completed 
by 1968 primary pupils and 1034 secondary pupils. 

The numbers of pupils who took both the Renaissance Learning Star 
Reading test and the Suffolk Reading Scale 2 tests were close to the 
required number and provided a good basis for the analyses; this was also 
the case for the Renaissance Learning Star Mathematics test and the 
Progress in Mathematics tests. 

A strong correlation was established between Star Reading and the 
Suffolk Reading Scale 2 tests (Pearson correlation coefficient of 0.91). 

Star Reading scores were related to reading ages derived from the Suffolk 
Reading Scale 2, providing an equating graph for comparative purposes. 

Star Mathematics and Progress in Mathematics (PiM) tests were shown to 
correlate reasonably strongly (correlation coefficients ranging from 0.58 to 
0.75 for different PiM tests). 

Star tests and English national curriculum teacher assessment levels 
correlated well (0.85 for reading and 0.81 for mathematics), particularly in 
view of the short length of the tests. 

Star Mathematics was equated to the English national curriculum level 
equivalents given in Progress in Mathematics; the correlation was good 
(0.84). 

The strong correlations provide evidence that both Star Reading and Star 
Mathematics are suitable for use in England. 

None of the results should be regarded as absolute; the appropriate 
sections of the report indicate any caveats. 




2 Introduction 

This is the report of the empirical equating study carried out as part of the launch of 
Renaissance Learning products in the UK, for which a variety of validation measures 
were required. 

2.1 Aims 

According to current thinking 1 , the validation of a test consists of a systematic 
investigation of the claims that are being made for it. In the case of Star Reading and 
Star Mathematics, the claims are that the tests give an accurate and useful indication 
of students’ reading and mathematical attainment. The tests are in wide use in the US, 
but the UK launch of the products required evidence that these claims could be made 
in the new educational context. 

One well-established method of validation is the collection of ‘concurrent’ evidence. 
If scores on the tests to be validated correlate highly with established tests that have 
an already validated claim, this correlation is evidence for the validity of the new test. 
This equating study therefore aimed to collect concurrent validation evidence by 
testing the same students on Star Reading and Star Mathematics and on tests or 
assessments of the same subject-matter, already established in the UK. The hoped-for 
outcomes were: 

• A substantial correlation between data from Star Reading and an established UK 
reading test, demonstrating that Star Reading tests the same construct as the UK 
test 

• An equating graph showing the relationship of Star Reading scores to established 
reading ages, in order to include UK reading ages as an additional outcome for 
the Star tests 



1 See, for example: American Educational Research Association, American Psychological Association 
and National Council on Measurement in Education (1999). Standards for Educational and 
Psychological Testing. Washington, DC: AERA. 
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• A substantial correlation between data from Star Mathematics and an established 
UK mathematics test, demonstrating that Star Mathematics tests the same 
construct as the UK test 

• A good correlation between the Star tests and England national curriculum 
teacher assessment levels, together with cut-scores on the RL ability scale. A 
good correlation would demonstrate that the Star tests are relevant to the 
England curriculum. (As the national curriculum levels are defined broadly, 
including aspects such as practical mathematics, full coverage cannot be expected 
in a short test.) 

• A good correlation between the Star Mathematics tests and the national 
curriculum level equivalents given in the Progress in Mathematics tests; aligned 
with these, equivalent cut-scores on the RL ability scale. 

2.2 Methodology 

The Renaissance Learning tests concerned were the computer-administered adaptive 
tests: Star Reading; and Star Mathematics. The UK paper-based tests were selected on 
the basis of being well regarded and widely used, as well as relatively recently 
standardised for the relevant age group (from age 6 to 14 years). These were: 

• Progress in Mathematics 6-14, by Tandi Clausen-May, Hanna Vappula and 
Graham Ruddock of NFER, published by nferNelson, standardised in 2003 

• Suffolk Reading Scale 2, by Fred Hagley, published by nferNelson, standardised 
by NFER in 2001. 

There is one test for each year group in the Progress in Mathematics series, with no 
overlapping items, but the Suffolk Reading Scale 2 comprises three levels for the age 
range, with some shared items. Primary pupils took Level 1 and 2 tests and secondary 
pupils took Level 3 tests. For mathematics, pupils took the age-appropriate test. 

Pupils were required to take the computer-administered and paper-based tests within a 
two week period, initially designated as 2 nd to 13 th October 2006. This testing period 
was extended because schools reported various difficulties and therefore failed to 
complete all the tests with pupils within the time available. In order to compensate 




for this shortfall, an additional three schools were added to the original sample and the 
time frame was extended to the first week in December. The final parcel of completed 
tests was delivered to the NFER in the second week of December. 

Background information on the pupils involved was also collected. This included date 
of birth, gender, home language, special educational needs and eligibility for free 
school meals, a proxy measure of social deprivation. The latter variable is statistically 
associated with lower attainment. This information was gathered by means of a pupil 
data form (pdf) that was placed on the Renaissance Learning website. The final school 
returned this data in mid- January. 

2.3 The samples 

Samples for the study were provided by Renaissance Learning from schools known to 
be using their products. The samples required were specified as 450 pupils for each 
test in each of the relevant age groups (year 2 to year 9 inclusive). In order to allow 
for this number, materials for 500 pupils were provided for each test and year group. 

The original plan was that Renaissance Learning would select 20 primary and 20 
secondary schools to take part, testing both reading and mathematics. In the event, 
recruiting this number of schools proved more difficult than Renaissance Learning 
had anticipated and the trial took place with fewer schools in the initial sample. This 
number was further reduced as some schools declined to participate, a situation that 
did not become apparent until well into the testing period. 

Tables 2.1 to 2.3 give details of materials sent out and completed, by test and subject. 
The details of the mathematics tests are divided according to primary (6-11) or 
secondary (12-14) age range. 
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Table 2.1: Primary mathematics (Progress in Mathematics 6-10) 



Year group 


2 


3 


4 


5 


6 


Age 


6-7 


7-8 


8-9 


9-10 


10-11 


Test 


PiM6 


PiM7 


PiM8 


PiM9 


PiMlO 


Number of schools in sample 


17 


17 


17 


17 


17 


Schools allocated materials 


12 


13 


14 


14 


13 


Schools returning used materials 


12 


13 


14 


14 


13 


Schools not submitting pupil data 


1 


1 


1 


1 


1 


Number of pupils allocated tests 


364 


404 


458 


462 


385 


Number of completed tests returned 


352 


394 


434 


453 


373 



Table 2.2: Secondary mathematics 


(Progress in Mathematics 11-13) 


Year group 


7 


8 


9 


Age 


11-12 


12-13 


13-14 


Test 


PiMll 


PiM12 


PiM13 


Number of schools in sample 


20 


20 


20 


Schools allocated materials 


12 


13 


11 


Schools returning used materials 


11 


12 


10 


Schools not submitting pupil data 


5 


5 


5 


Number of pupils allocated tests 


364 


476 


372 


Number of completed tests returned 


287 


341 


255 
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Table 2.3: Reading (Suffolk Reading Scale 2) 



Year group 


2-3 


4-6 


7-9 


Age 


6-8 


8-11 


11-14 


Test 


SRS 


SRS 


SRS 




LI 


L2 


L3 


Number of schools in sample 


17 


17 


20 


Schools allocated materials 


15 


14 


15 


Schools returning used materials 


15 


14 


15 


Schools not submitting pupil data 


1 


1 


5 


Number of pupils allocated tests 


764 


1,305 


2,069 


Number of completed tests returned 


713 


1,255 


1,034 



All the schools were in England, rather than other parts of the UK. The samples of 
schools were never expected to be representative of all schools in England, but the 
two were nevertheless compared to the national population. Because of the small 
numbers of schools involved, it was not possible to establish if the differences were 
significant. In primary schools, there was a greater representation of schools from the 
Midlands and those in the lowest band of key stage 2 attainment in the sample. 
Additionally, there was a much higher representation of schools from London 
Boroughs than in the national population. In terms of secondary schools in the sample, 
a higher percentage came from the Midlands, but rather more of these came from the 
middle and second highest bands of key stage 3 attainment for reading. For 
mathematics, nearly half the secondary schools were in the middle band of attainment 
for key stage 3. 
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3 Outcomes 

Progress in Mathematics (PiM) tests were marked and checked by NFER researchers 
and the data was entered by the NFER’s Database Production Group. The data from 
the Suffolk Reading Scale 2 tests (SRS) were entered directly as no marking was 
required. 

This test level data was then matched to background data. A further match was made 
to pupils’ Rasch ability score for the Star Reading and Star Mathematics tests. The 
latter data was provided by Renaissance Learning (RL). Both RL and NFER staff 
made great efforts to persuade schools to complete all the background data, but in 
spite of this, some pupils had incomplete data and were therefore excluded from the 
final analyses. 

3.1 Teacher assessment levels 

Teacher assessment levels give an indication of the spread of attainment of the pupils 
in the sample. They are based on the national curriculum in England and are therefore 
not applicable to the UK as a whole, where four different education systems are in 
operation (England, Scotland, Wales and Northern Ireland). The levels used in this 
analysis were derived from the pupil data forms. The numbers of pupils at each level 
for reading (for those pupils for whom data was available) are shown in Table 3.1. 




10 




Table 3.1 Teacher assessment levels for reading 





Frequency 


Percent 


Cumulative Percent 


Working towards 
Level 1 


49 


1.8 


1.8 


Level 1 


239 


9.0 


10.9 


Level 2 


671 


25.3 


36.2 


Level 3 


537 


20.3 


56.5 


Level 4 


567 


21.4 


77.8 


Level 5 


480 


18.1 


96.0 


Level 6 


92 


3.5 


99.4 


Level 7 


12 


0.5 


99.9 


Level 8 


3 


0.1 


100.0 


Total 


2650 


100.0 




As can be seen in the table, there were substantial numbers of pupils between Levels 2 
and 5, with enough for analyses at Level 1 and 6. Any results relating to Levels 7 and 
8 should be treated with considerable caution because of the low numbers of pupils. 


The teacher assessment levels for mathematics are shown in Table 3.2. 


Table 3.2 Teacher assessment levels for mathematics 




Frequency 


Percent 


Cumulative Percent 


Working towards 
Level 1 


28 


1.0 


1.0 


Level 1 


197 


7.3 


8.3 


Level 2 


864 


31.8 


40.1 


Level 3 


604 


22.2 


62.3 


Level 4 


466 


17.2 


79.5 


Level 5 


379 


14.0 


93.4 


Level 6 


136 


5.0 


98.5 


Level 7 


39 


1.4 


99.9 


Level 8 


3 


0.1 


100.0 


Total 


2716 


100.0 


2716 



As for reading, there were substantial numbers of pupils at levels 2 to 5 , with enough 
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for analysis at levels 1 and 6. The low numbers of pupils at levels 7 and 8 indicate that 
any results for these levels should be treated with caution. 



3.2 Test scores 

3.2.1 Suffolk Reading Scale test scores 

Pupils’ mean scores on the three levels of the Suffolk Reading Scale 2 are shown in 
Table 3.3. The table also shows the numbers of pupils in the sample. 



Table 3.3: Mean scores in Suffolk Reading Scale tests 





SRS1A 


SRS2A 


SRS3A 


Marks available 


75 


86 


76 


Mean score 


35.1 


50.0 


48.4 


Median 


37 


52 


49 


Standard deviation 


17.0 


13.5 


11.0 


Mark range 


0-71 


6-80 


2-72 


Age standardised score - mean 


96.3 


99.3 


100.4 


- standard deviation 


15.3 


14.8 


13.7 


Number 


713 


1255 


926 


Number boys / girls 


337/371 


633 / 600 


376 /499 



The age standardised scores give an indication of the overall attainment of the sample, 
as the nationally standardised mean is set to 100, with a standard deviation of 15. This 
shows that the samples were very close to average attainment overall, with the sample 
for SRS1 being slightly lower-attaining than the national average. Boys and girls were 
fairly evenly represented in SRS1 and SRS2, with a higher proportion of girls in 
SRS3. There was no statistically significant difference between their mean scores, 
which were very close to the mean score for the relevant test. 

Pupils’ performance was also analysed by the ‘free school meal’ variable. The mean 
scores of pupils identified by this variable were consistently lower than those of the 
rest of the samples; the difference was statistically significant for SRS1 and SRS2, but 
there was no significant difference between the mean scores of these groups for SRS3. 

As would be expected, scores by teacher assessment levels increased level by level 
with mean scores showing statistically significant differences in most cases. 
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Where the differences were non- significant, the levels were at the extremes for the 
age group. 



The reliability coefficient (Cronbach’s Alpha) was very good for all three tests, 
ranging from 0.97 for SRS1 to 0.92 for SRS3, showing that the tests were internally 
consistent and were measuring the same construct - something to be expected in 
established tests of this nature. 

3.2.2 Progress in Mathematics test scores 

Pupils’ mean scores on the Progress in Mathematics tests are shown in Table 3.4. The 
table also shows the numbers of pupils in the sample. 



Table 3.4: Mean scores in Progress in Mathematics tests 





PiM 


PiM 


PiM 


PiM 


PiM 


PiM 


PiM 


PiM 




6 


7 


8 


9 


10 


11 


12 


13 


Marks available 


28 


28 


35 


45 


45 


50 


50 


50 


Mean score 


18.7 


17.2 


21.7 


25.3 


24.8 


31.7 


27.9 


23.0 


Median 


20 


18 


23.5 


26 


25 


35 


28 


22 


Standard deviation 


6.0 


5.7 


8.4 


10.2 


10.5 


11.8 


10.6 


11.6 


Mark range 


2-28 


2-28 


0-35 


3-44 


1-45 


1-50 


0-49 


1-50 


Age standardised score - 
mean 


92.7 


90.8 


96.9 


96.6 


95.2 


95.3 


100.9 


98.7 


- standard deviation 


16.2 


14.2 


16.4 


15.4 


14.4 


14.2 


14.2 


12.6 


Number 


352 


394 


434 


453 


373 


287 


341 


255 


Number: boys 


170 


180 


206 


242 


194 


134 


141 


113 


girls 


175 


206 


217 


202 


176 


151 


199 


142 



The age standardised scores show that the pupils in the primary age groups were, on 
average, rather low attaining, whereas the secondary pupils were close to the national 
average. There were more girls than boys in the sample overall, with girls 
outnumbering boys by between five and 58 cases on all tests with the exception of 
PiM 9 and 10 (for pupils age 9-11). There was no statistically significant difference in 
their mean scores apart from the highest test, PiM 13, where the higher mean score for 
boys (25.8) was significantly different from that of girls (20.8) at the one per cent 
level of significance (highly significant in statistical terms). 
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As for the reading tests, pupils’ performance was also analysed by the ‘free school 
meal’ variable. There was a statistically significant difference between the mean 
scores of those pupils receiving free school meals and other pupils for all the tests 
apart from PiM 12 and 13. The level of significance varied, with PiM 7, 10 and 11 
showing the most significant difference. 

As would be expected, scores by teacher assessment levels increased level by level 
with mean scores showing statistically significant differences in most cases. 

The reliability coefficient (Cronbach’s Alpha) was very good for all tests, ranging 
from 0.87 for PiM 7 to 0.94 for PiM 1 1 and 13. 
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4 Equating 

4.1 Equating reading 

4.1.1 Scores 

To equate the RL and the Suffolk Reading scale in terms of ability, two methods were 
used: the first equated raw scores on the Suffolk Reading Scale to the RL Star 
Reading tests. The graphs relating to this model are Figures 1.1 to 1.3 in Appendix 1. 
For any given score on the test on one axis, the equivalent score on the other test can 
be read off from the other axis. The broken lines on the graph are the 95 per cent 
confidence intervals, representing a five per cent ‘margin of error’ for the equating 
line. These graphs show that for SRS1, the confidence intervals are quite narrow for 
the range of scores, whereas for SRS2 and SRS3 the confidence bands are much wider 
at lower scores. The narrower the confidence interval, the greater is the reliability of 
the equating findings. Individually, the correlation coefficient of each of these with 
the RL Star Reading scale was good: for SRS1 it was 0.84, for SRS2 it was 0.88 and 
for SRS3 it was 0.78. 

The second method involved two stages: The Suffolk raw scores were equated to a 
Suffolk ability scale using a 2-parameter IRT model. This was possible because there 
were common items in the different tests. The resulting graphs, by test, are Figures 
1.4 to 1.6 in Appendix 1. This ability scale was then equated to the RL Star Reading 
ability scale. The resulting graph can be seen in Figure 1.7 in Appendix 1. This 
method of equating would seem to be more reliable as it makes use of the whole 
sample measure of reading ability, whereas the alternative one stage model is done for 
each of the three Suffolk tests. Using the evidence from this graph, where zero on the 
Suffolk scale equates to greater than zero on the RL scale, it would seem that the RL 
Star Reading test is designed for slightly less able pupils overall and is certainly 
appropriate for the pupils tested. It should be noted that the width of the confidence 
intervals at the top and bottom of the graph indicate that there is more uncertainty in 
these areas. The overall correlation of the Suffolk ability scale to the RL ability scale 
was 0.91, a high figure which shows that both tests are likely to be concerned with a 
similar construct of reading. 
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4.1.2 Reading ages 

Direct equipercentile equating was used to link SRS reading ages to the RL Star 
reading tests. The resulting graph can be seen in Figure 1.8 in Appendix 1. Reading 
ages above 192 months (16 years) have been excluded as suggested in the Teacher’s 
Guides that accompany the tests. 

The graph shows that although both scales increase together, they do not increase at a 
uniform rate. In spite of this, reading ages can be read off the graph for any Star 
reading ability. For example, a RL ability of 0 would give a reading age of 104 
months (8 years 8 months). A score of 2 on the RL scale would give a reading age of 
131 months (10 years 11 months). 

When using reading ages, it should be noted that there is some uncertainty in the 
original reading ages given in the Suffolk Reading Scales because reading ages are 
derived from the standardised score calculations and represent a less reliable metric 
than standardised scores. This uncertainty is likely to be compounded by using the 
data for a further level of equating. Because of this uncertainty in the measure, the 
reading ages should be used with care. 
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4.1.3 Teacher assessment levels 



Teacher assessment levels were used to equate to the RL Reading Rasch Ability scale, 
resulting in cut scores as shown in Table 4.1. This table is expressed as cut score, i.e. 
the lowest score for the level. In this table, RL scores of -5.39 to -3.48 represent the 
range within Level 1 . 

Table 4.1 Teacher assessment levels related to RL Rasch Ability scale 



TA level 


Approximate RL Star 
Reading cut score 


Number 


Level 1 


-5.39 


183 


Level 2 


-3.47 


574 


Level 3 


-0.25 


483 


Level 4 


1.30 


524 


Level 5 


2.68 


433 


Level 6 


4.45 


86 


Level 7 


5.29 


12 


Level 8 


5.76 


3 



The RL Star Reading test correlated well with teacher assessment levels, with a 
coefficient of 0.85 based on 2324 pupils. The cut scores for Levels 7 and 8 should be 
treated with caution because of the very small numbers and it should also be borne in 
mind that teacher assessment levels are not subject to moderation by other teachers or 
external bodies. For the purposes of this analysis, the ordinal variable representing 
teacher assessment levels has been treated as though the intervals were of equivalent 
size. This should add to the caution with which the results are treated. 
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4.2 Equating mathematics 

4.2.1 Correlations 

The Progress in Mathematics tests, unlike the Suffolk Reading Scale tests, do not have 
any common items. The raw scores for the tests, therefore, have been mapped to the 
RL scale individually. The graphs showing these relationships are Figures 2.1 to 2.8 
in Appendix 2. The correlations relating to these graphs are shown in Table 4.2. 

Table 4.2 Pearson correlation coefficients for PiM and RL Rasch Ability 
scale 



Test 


Coefficient 


Number 


PiM 6 


0.58 


290 


PiM 7 


0.73 


297 


PiM 8 


0.74 


403 


PiM 9 


0.74 


415 


PiM 10 


0.75 


354 


PiM 1 1 


0.74 


257 


PiM 12 


0.70 


311 


PiM 13 


0.73 


233 



The equating trials show a reasonable correlation between performance on most tests, 
(0.70 or higher), with a lower value for PiM 6 (0.58). For the higher correlations, this 
means that one score explains 50 per cent or more of the variation of the other score. 
Figure 2.1 in Appendix 2 demonstrates the wide confidence bands around the 
equating, indicating that the relationship is not secure. It should be noted that there is 
similar uncertainty at the lower ability end of PiM 9 and more generally with the tests 
taken by secondary pupils (PiM 11-13). 

The slope of the graphs for pupils aged 6 to 8 (PiM 6 and 7) are similar in their central 
sections, suggesting that improvement on one scale is matched by similar 
improvement on the other. PiM 8 to 1 1 - based on their central sections - suggest that 
scores on Progress in Mathematics tests increase more rapidly than the increase on the 
RL ability scale for Star Mathematics. The relationship between the two tests at the 
upper end of the age band (PiM 12 and 13) is somewhat erratic, with varying 
confidence bands. It is possible that some of these tests are focusing on slightly 
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different constructs of mathematical skills. However, the smaller sample size has also 
affected these extremes in particular. 



4.2.2 Teacher assessment levels 

Teacher assessment levels were used to equate to the RL Mathematics Rasch Ability 
scale, resulting in cut scores as shown in Table 4.3. 

Table 4.3 Teacher assessment levels related to RL Rasch Ability scale 



TA level 


Approximate RL Star 
Mathematics cut score 


Number 


Level 1 


-6.44 


157 


Level 2 


-4.30 


757 


Level 3 


-1.70 


576 


Level 4 


-0.49 


438 


Level 5 


0.52 


348 


Level 6 


1.73 


125 


Level 7 


2.48 


38 


Level 8 


3.74 


3 



The RL Star Mathematics test correlated well with teacher assessment levels, with a 
coefficient of 0.81 based on 2460 pupils. As discussed above, the cut scores for 
Levels 7 and 8 should be treated with caution and it should also be borne in mind that 
teacher assessment levels are not subject to moderation by other teachers or external 
bodies. Also, as indicated for the levels in the reading tests, the ordinal variable that 
represents teacher assessment levels has been treated as though the intervals were of 
equivalent size, even if they are not. 

4.2.3 PiM national curriculum levels 

The Progress in Mathematics tests provide a link to English national curriculum 
levels, although this is not provided for the Suffolk Reading Scale tests. The 
Teacher’s Guides emphasise that a relatively short assessment such as the Progress in 
Mathematics tests can only offer an indication of the pupil’s probable level of 
attainment. These levels should therefore be regarded as a contribution to teachers’ 
overall judgments. 
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The national curriculum level cut scores and the number of pupils at each level can be 
seen in Table 4.4. These values are very similar, though not identical, to the teacher 
assessment results in Table 4.3, giving support that both measures are reasonably 
accurate. 

Table 4.4 PiM National curriculum levels related to RL Rasch Ability 
scale 



National 
curriculum level 


Approximate RL Star 
Mathematics cut score 


Number 


Level 1 


-6.00 


130 


Level 2 


-4.21 


559 


Level 3 


-1.83 


610 


Level 4 


-0.39 


491 


Level 5 


1.14 


265 


Level 6 


2.18 


53 



The figures do not include pupils whose NC level was ambiguous from the Teacher 
Guide (eg ‘2c and below’), but the numbers were adequate for the analysis. The 
correlation between the PiM national curriculum levels and the RL ability scale for 
mathematics was based on 2131 pupils and was a good value (0.84). 
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5 Conclusions 

Star Reading and Star Mathematics correlate highly with established UK tests of 
reading and mathematics, demonstrating concurrent evidence of their validity for use 
in this country. 

Star Reading and Star Mathematics also correlate well with teacher assessments of 
English national curriculum levels, and in the case of Star Mathematics also with 
published national curriculum levels, demonstrating that they are likely to test a 
substantial subset of national curriculum skills and understandings. 
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Appendix 1 : Reading equating 
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Appendix 2: Mathematics equating 
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